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ABSTRACT 

A review of more than 3, COO books, monographs, 
journal articles, papers and other reports was conducted to discover 
what )cinds of design, measurement, and anc.lytical lessons have been 
learned as a result cf the enormous volume of 

college-impact-on-student studies that have been published in the 
last 20 years and to determine how these lessons might help 
researchers in this area to be more effective. Eight "lessons" are 
discussed which are meant to offer a constructive critique of current 
research designs, methods, and substantive foci; in addition, a 
constructive critique is offerred suggesting ways in which that 
research might be made more rigorous, informative, and supportive of 
educational progrfun and policy decision making. It is suggested that 
greater efforts be made in the following eight tasks, which ace 
examined in detail: (1) differentiating between simple charge 
occurring in students during the college years and the "net effects" 
of college, (2) better estimating the magnitudes of college effects, 
(3) better examining when student change occurs, (4) better 
exploration into measuring indirect as well as direct collegiate 
effects, (5) more focus on college's effect coming about because of 
specific student characteristics (race, age, etc.), (6) making 
greater use of qualitative research methou??, (7) expanding the 
theoretical perspectives that guide research and assessment study 
designs, and (8) greater focus on the effects of the academic program 
and the teaching-learning process on students overall. Contains 10 
references. (GLR) 



* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 



35 



CO 

(=) 

W TWENTY YEARS OF RESF'VRCH ON COLLEGE STUDENTS: 

LESSONS FOR FUTURE RESEARCH 



by 



Patrick T. Terenzini 
Professor and Senior Scientist 
Center for the Study of Higher Education 
133 Willard Hall 
The Pennsylvania State University 
University Park, PA 16802 
Telephone: 814-865-6346 



and 



I 

4 



U t Df MfVTMCNT Of EDUCATION 
ON«ct (X Educat«on«i RtMtrch tnd Improvtmtnt 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIO 

aihi> document htt b«in reproduced •» 
received from the perior^ of ofgenizitior 
oftgiruting >t 

O Mmor chengei heve beer^ mede to tmprove 
reproduction queltty 

e Pwnti uf vtev^ Of optnton* »tited m tht»docu 
ment do not necewirtiy represent otf»ctel 
OERI petition or poltry 



Ernest T. Pascarella 
Professor of Educational Psychology 
College of Education (M/C 147) 
University of Illinois — Chicago 
Chicago, IL 60680 
Telephone: 312-996-8131 



BEST COPY AVAiLABLE 



"PERMISSION TO REPRODUL.. THIS 
MATERIAL HAS BEEN GRANTJ=D BY 



AIR 

TO THE EDUCATIONAL RESOURCES 
INFORMATION CEN FER {ERICj " 



Paper presented at the annual Forum of thc^ Association for Institutional 
Research, Louisville, KY, May 1990. 



ERIC 



2 



foriimmgmmntRmmrch, Policy Armfytis, and Planning 



This paper was presented at the Thirtieth Annual 
Fcrum of the Association for Institutional Research 
held at The Gait House* Louisville > Kentucky, 
May 13-16, 1990 This paper was reviewed by the 
AIR Forum Publications Committee and was judged to 
be of high quality and of Interest to others 
concerned with the research of higher education. 
It has therefore been selected to be Included In the 
ERIC Collection of Fortm Papers. 

Jean Endo 
Chair and Editor 
Forum Publications Editorial 
Advisory Committee 



ERIC 



8 



TWENTY YEARS OF RESEARCH ON COLLEGE STUDENTS: 
LESSONS FOR FUTURE RESEARCH 

Twenty years have passed since Feldman and Newcomb (1969) published 
their landmark book, The Impact of College on Students , in which they 
reviewed forty years of theoretical propositions and empirical researc.i 
on college students* Much has happened since then as the effects of 
college on students as an field of inquiry has grown qualitatively and 
quantitatively ir both theory and method. Whole areas of inquiry, such 
as the effects of college on learning, cognitive development, moral 
reasoning and various indices of status attainment, have developed into 
maturity. Theories of student development and change have emerged in 
sometimes daunting number and variety. Multivariate statistical 
procedures, adequate to the task of testing and extending these emergent 
and complex theories, have became increasingly accessible to scholars (a 
development with mixed blessings). History-altering advances in both 
mainframe and microcomputing hardware and software have been the 
handmaidens of these advances, facilitating both the complex statistical 
analyses testing equally complex theories and permitting the analysis of 
large, nationally-representative databases. 

In the last decade, the increasing costs of college attendance and 
operation, as well as mounting criticism of the quality of undergraduate 
education in the United States, have also spurred research on coV' 
outcomes. Earlier questions of cost, now answered, have been followed by 
questions of worth and value, of education's "return on investment" in 
both economic and noneconomic terms* "Assessment" of undergraduate 
student learning gained in popularity as a vehicle for a public 
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accounting of an institution's stewardship of its resources and as a 
mechanism for iir4)roving the quality of the education offered. 

The phenomenally productive conjunction of all these developments — 
in theory, methodological sophistication, computing, cos^, criticism, and 
oversight — created a burgeoning literature on the effects of college on 
students that has more thcin doubled in size that produced in the 
preceding four decades reviewed by Feldman and Newcomb (1969). What have 
we learned about studying college Impacts from this enormous volume of 
research? What kinds of design, measturement , and analytical lessons can 
be drawn from it that might help researchers do better research in the 
future? This paper discusses eight such "lessons." It offers a 
hopefully constructive critique of current research designs, methods, and 
substantive foci, suggesting ways in which that research might be made 
more rigorous, informative, and supportive of educational program and 
policy decision-making. 

METHODS 

Virtually all research relevant to student change during college and 
produced sincj 1967 was reviewed. To identify relevant studies, searches 
were made of various abstracting dociMients or latabases, such as 
Sociological Abstracts , Psychological Abstracts / Sociology of Education 
Abstracts , Dissertation Abstracts , College Student Personnel Abstracts # 
Higher Education Abstracts , and the ERIC system. Recent conference 
proceedings from such scholarly and professional associations as the 
American Educational Research Association, Association for the Study of 
Higher Education, and the Association for Institutional Research were 
also reviewed. Finally, an extensive network of colleagues was used to 
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obtain unpublished papers or technical reports which dealt with college 
impact . 

Despite the potential advantages of meta-analysis for synthasizing a 
large body of literature (see Cooper^ 1932; Glass, 1977; Jackson, 1980; 
Pillemer & Light, :I980; Light & Pillemer, 1982), a number of factors led 
us to decide against using meta-analysis. First, meta-analytic 
techniques have come under close and often critical scrutiny in terms of 
their producing a truly objective and meaningful synthesis of evidence. 
These and related criticisms of meta-analysis are cogently reviewed by 
Slavin (1984), who concludes that meta-analysis can be a useful 
supplement to traditional narrative, explanatory reviews, but should not 
be seen as a replacement for them. 

The second, and perhaps more irnportant, reason for our deciding 
against meta-analysis as the primary method of synthesizing the evidence 
was siitqply the remarkable diversity of ways in which research on the 
impact of college on students is reported. The single fact is that in 
many areas of inquiry the bioad range of statistical evidence employed to 
report results makes the use of quantitative synthesis inpractical if not 
impossible. Related to this issue was our concern that the requirements 
of quantifying study results in a comparable metric would exclude studies 
based on naturalistic inquiry or other relevant investigations where the 
recults were sijt?>ly not amenable to the computation of effect sizes. 

Thus, we turned to a narrative explanatory synthesis as our primary 
approach to the analysis of evidence. In this approach we were guided by 
the criterion of "weight of evidence." That is, given a logical analysis 
of the studies conducted, what does the weight of evidence suggest about 



the influence cj college or the influence of different aspects of the 
collegiate experience? 

BINDINGS AND CONCLUSIONS 

Eight lessons can be drawn from this review of more than 3,000 

books, monographs, journcd articles, papers, and other research reports: 

1. Simple change must be differentiated from the "net effects" of 
college . 

The vast majority of studies we reviewed focus on change during the 
college years. Some of these studies were cross-sectional in design, 
based on a random sanpie of enrolled students (sometimes all class years, 
sometimes only seniors and freshmen). While the class year variable in 
cross-sectional studies can be argued to be a proxy for varying amounts 
of exposure to college, this design has numerous flaws which threaten its 
internal validity (see, for example, Pascaiella, 1987; Pascarella & 
Terenzini, in press. Appendix A). Other studies were longitudinal panel 
studies, the same individu^,ls in a cohort of entering freshmen being 
followed-up one or mere years later. Longitudinal designs using only 
college students, while generally much stronger fiv^signs for mea£\iring 
change, shed no light on the extent to which any observed changes are due 
to the college experience since the degree of exposure to college is the 
same for all study participants. In over twenty years of research, the 
number of studies we found that employed a control group of college-age 
individuals who did not attend college can be counted on the fingers of 
one hand. 

In short, virtually all of the studies done to-date shed useful 
light on the extent to which students change during the college years , 
but change during college is not the same as change due to college. The 



drop-off in the volume of relevant research when one moves from studies 
of change during college to studies of change due to college attendance 
is striking and should be a source of some concerri. 

Change due to the college experience is sometimes referred to as the 
"net effects" of college. This phrase refers to the changes in students 
over time that ccui be attributed to the college experience and not to 
other sources of influence, such as normal maturation or conditions and 
events external to the campus. For exanple, if one is interested in 
assessing the extent to which increases in cognitive development can be 
attributed to col.lege attendance (versus non-attendance), one must take 
into account those other variables besides college attendance which are 
likely to influence changes in critical thinking (e.g., intelligence, 
academic aptitude and high school achievement, cognitive development, 
socioeconomic status, and so on). If one were to compute the association 
between college attendance and a measure of cognitive development while 
statistically controlling for intelligence, the results would be an 
estimate of the effect of college on cognitive development net of (or 
independent of) the confounding influence of initial intelligence. 

It is essential to the design of effective programs, professional 
integrity, and public credibility that claims about the benefits of 
college attendance be supportable with evidence that separates college 
effects from non-college influences ve.g., pre-college differences and 
normal maturation). Current claims about the benefits of college 
attendance frequently extend well beyond the empirical evidence to 
support them. Controlling the numerous alternative, non-college sources 
of influence can be a daunting undertaking. It will require greater use 
of non-college control groups and more extensive use of relevemt theories 



in the design of studies- Such careful theoretical preparation and 

grounding is not one of the distinctive characteristics of most of the 

research done over the past two decades ^ but higher education as a field 

of inquiry has clearly started down that road, and we wish to encourage 

its continuation. Theory-based research will not only be more sharply 

focused and parsimonious, but it is also likely to reflect more fully the 

complexity of college impacts- 

2. Effect sizes should be estimated . 

Most studies identify statistically significant changes in students 

over time, but few examine the magnitudes of those changes- While it may 

be meaningful to report siiTiply whether an independent variable is related 

to a dependent, outcome variable at some level of statistical 

significance, it is much more meaningful, as well as theorecically and 

practically informative, also to estimate the strength of the relation. 

Many of the studies we reviewed failed to report even the most 

rudimentary information (e.g-, means and standard deviations) that might 

1 

be used to estimate effect sizes - 

Reporting estimated effect size can reasonatdy be expected to lead 
to theories that are more parsimonious and better reflect the reality of 
college impact. It is necessary, but not sufficient, to know that group 
differences or changes are not due to chance- It is equally important to 
know whether the difference or change is educationally or 
a dhunistratively significant. Administrators can be expected to make 
programmatic, policy, and budgetary decisions at least in part on 
outcomes assessment information. They have a reasonable right to know — 
and institutional researchers a corresponding obligation to provide 
information on — whether the impact of college or some aspect rf the 
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college experience is large enough to warrant attention, resources, and 
action. 

3. Little is known abut when changes occur . 

Much of the assessment literature examines change during the 
freshman year, or from freshman to senior years but not during the 
intervening years. Only a handful of the studies we reviewed monitored 
change on an annual, sequential basis. Thusr, we know little about 
whether change is mostly linear and monotonic, or whether it is primarily 
episodic and discontinuous over the college years. Moreover, it seems 
reasonable to suggest (and there is some basis for believing) that the 
pacing of change varies across outcome areas. Much more attention needs 
to be given to the analysis of the timing of change during college. For 
policy and program planning purposes, it is, of course, in^rtant to know 
whether change occurs, but it is at least as in^rtcuit to know when an 
interventior. will make a difference and when it won't so that 
institutional efforts and resources can be brought to bear when they are 
most likely to be effective. 

4. Important indirect college effects may be going unnoticed . 

Much of rhe research on colleges' net effects indicates they cend to 
be small in size. That college effects are, in fact^ small is only one 
of the possible explanations for this fairly consistent finding. Others 
incluQu weak or unreliable measures; use of distal predictor measures 
(e.g., living on- vs. off-campus) instead of more proximal ones (e.g., 
measures of the frequency and nature of students' interactions with peers 
and faculty members within the living environment); measures constructed 
to place a premium on test-retest reliability (cind therefore perhaps 
biased against showing change) (Winter^ 1979; Winter, McClelland, & 
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Stewart, 1981), and the essentially conservative nature of the analytical 
procedures typically employed (e.g., hierarchical regression or path 
analytic models, which attribute variance jointly explained by 
pre-college and college experience variables entirely to the pre-college 
variable set). 

Another possible explanation — often overlooked — is that a 
college effect may be indirect, as well as direct. A direct effect can 
be thought of as the unmediated influence of one ^^ariable on another 
(i.e., the impact is direct and does not pass through an intervening 
variable). Although the descriptor "direct" is seldom used in the 
research literature, direct effects are by far the most frequently 
estimated effects in educational or social science research. Using our 
previous example, if going to college has a significant association with 
cognitive development when intelligence is controlled, then it can be 
said to have a direct effect on cognitive development net of 
intelligence. Conversely, if the association between college attendance 
is nonsignificant when intelligence is taken into account then college 
can be said to have no direct effect on cognitive development net of 
intelligence. 

Although it is seldom estimated in the existing research, a variable 
may also have an "indirect" or mediated effect on an outcome. This 
occurs when that variable's effect is transmitted through an intervening 
variable(s). For example, it is possible that college attendance may 
have an important indirect effect on adult cognitive development by 
influencing a person's reading habits. Thus^ the path of indirect 
influence would be college attendance directly affecting reading habits^ 
and reading habits, in turn, directly affecting cognitive development. 
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In this and similar ways college could have a significant impact on a 
range of outcomes without having a direct effect on them. In our 
synthesis of the research evidence, we have been impressed by how many of 
the effects of college are, or could be, indirect. For example, while 
major levers of institutional influence (e.g., residence halls) lozY not 
have substemtial effects on student change in various areas, they do have 
important indirect effects, influencing other variables which, in turn, 
have a substantial impact on students. 

Thus, as much of the evidence we reviewed suggests, it is entirely 
possible that we may be underestimating or even misrepresenting the 
impact of many college influences by failing to consider their indirect 
effects. Because some source of influence in the causal chain is none 
step ren ^red from having a direct effect on a given outcome makes it no 
less theoretically or practically important. Indeed, its consideration 
may add substantially to our knowledge of educational effects. Of 
course, any consideration of indirect effects means that one must 
typically conceptualize research questions in terms of theoretical 
models, but such a process is likely to reflect more fully and acciirately 
the complexity of college iitpacts. 

5. Research on conditional effects will be increasingly important . 

A "general" effect suggests that a particular collegiate experience 
is the same for all students who experience it. A "conditional effect" 
suggests that the magnitude of the effect is conditional upon, or varies 
according to, the specific characteristics of the individuals being 
considered (e.g., minority vs. nonminority, male vs. female, 
traditional-aged vs. older students). Conditional effects are sometimes 
referred to as "interaction effects" in that individual student 
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dif f erences are said to "interact ' with the particular experience or 
condition thought to influence a particular outcome. 

Despite many undoubtedly sincere statements about tne importance of 
respecting individual student differences, relatively little attention 
has been given in the research and assessment literature t^ examining how 
college effects vary according to students* characteristics. Are the 
rfects of college the same regardless of the student's sex? Race or 
ethnicity? Academic ability? Socioeconomic status? Answers to such 
questions arc becoming increasingly important, particularly those related 
to race /ethnicity, sex, and age. If most demographic profiles of the 
college population for the next decad*^ are anywhere near accurate, 
minority and older students will constitute a much larger proportion than 
ths.'Y do at present. Current programs, designed primarily with 
traditional-aged white student populations in mind, may or may not have 
the same effects on different kinds of students. If more is known about 
how different kindr of students respond to similar experiences, programs 
and policies can be tailored and their effectiveness increased . 
6. Greater use should be made of qualitative research methods . 

The current literature on college ef fe -ts is almost exclusively 
quantitative. While the logical positivist, quantitative paradigm has 
served us well, judicious and creative qualitative approaches are capable 
of providing greater sensitivity to many of the subtle and fine-grained 
complexities of college impact than the more traditional quantitative 
approaches. Naturalistic and ethnographic inquiries may be particularly 
sensitive to the kinds of indirect and conditional effects just discussed 
We expect that in the next decade # iii?>ortant contributions to our 
understanding college impact will be produced by qualitative research 
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approaches. Forewarned is forearmed^ however: qualitative methods are 
often thought to be less rigorous and less demanding than quantitati/e 
approaches. They are not. 

7. A single paradigm dominates current research on college effects . 

Most of the prominent contributors to theory development and 
research on the impact of college or students have been or are 
psychologists. As a consequence, past and current research on college 
impacts is distinguished by its almost exclusive reliance on 
psychological models. Similarly, for the past quarter- century, graduate 
programs that have trained (and continue to train) many of higher 
education's researchers and administrators have had their conceptual 
origins largely in one theoretical genre. Theories from other fields 
have only recently begun to receive notice. Indeed, meny researchers in 
higher education appear to be unaware of a substantial theoretical and 
empirical literature relating to collegiate effects based in other 
disciplines, especially sociology and anthropology. The need for new 
perspectives on college impacts is particularly acute in studies of 
students' non-cognitive, psychosocial chemges. An alarming number of 
studies in these areas reflect little familiarity with the knowledge base 
outside the author's primary disfipline. Whether many of the observed 
changes reported in the literature are due to developmenx:al, psychosocial 
restructuring within students, or to learning through the socialization 
process of competencies, attitudes^ values, and behaviors valued by 
important others, remains very much an open and vital question. 
Single-paradigm research or assessment programs are likely to restrict 
the range of analytical vision, the depth of understanding, and, 
consequently, the effectiveness of academic and non-academic programs. 
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8, Certain areas of study are particularly in need of attention . 

While the sheer volume of studies of the effects of college on 
students done over the past twenty years is truly impressive, several 
important holes in the research fabric are identifiable and become more 
glaringly so as time passes. One in particular that stands out is the 
impact of the academic program and the teaching- learning process. How do 
different teaching and instructional approaches influence not only how 
much content is learned, but also what higher order thinking skills are 
developed? How and in what ways does the academic program influence 
values and personal development? Are there particular teaching or 
instructional approaches which are differentially effective for different 
kinds of students? Answers to these and similar questions will 
constitute major contributions to our understanding of the impact of 
college. 

A second important, but virtually unexplored, area of inquiry is the 
nature and dynamics of the collegiate experience for significant groups 
of nontraditional students. The absence of rigorous research on the 
effects of college on minority students and older students is 
particularly embarrassing to the higher education research community. As 
noted earlier, demographic forecasts for the next decade consistently 
indicate that the proportion of minority students in America's colleges 
and universities will increase, dramatically among some groups. Higher 
educational administrators cannot xong delay responding to this and 
related trends. If researchers are to help prepare our higher 
educational system for these changes, much, much more must be learned 
about how the collegiate experience, academic and non-academic, differs 
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for minority versus majorit/ groups, and for older versus 
traditional-aged students. 

Other areas of inquiry that have been largely ignored in any detail 
include the dynamics of student and faculty interaction, the corap^^-cative 
influences of faculty and peer groups, and institutional subenvironr'^'ts 
and subcultures that shape college's effects through their mediating 
influences on students' interactions with peers and faculty members. I^. 
will be particularly important to examine students' interpersonal 
experiences in both formal and informal learning settings. Continued 
rese£irch focusing on such distal environmental measures as living on- vs. 
off -campus, academic major, or similarly distal institutional 
characteristics, such as size, type of control, or even selectivity, is 
unlikely to advance our knowledge about college effects. More proximal 
and precise sipecif icdtion of the worlds in which students live are 
needed. 

SUMMARY 

This paper has attempted to help improve the quality of future 
student outcomes research on individual campuses by suggesting some of 
the future directions these studies might pursue. The maturation of this 
area of study over the years is appar*^:nt in studies of college iJipact 
published or reported in the last twenty years. Much remains to be done, 
however. In particular, we h^ve curgufcx-, future research and assessment 
studies and programs should devote greater efforts to eight t:isks: 1) to 
differentiating changes that occur during college from those that are due 
to college (so t>*at current claim« aocut the benefits of college 
attendance might be supported); 2) to estimating the magnitudes of 
college effects (so that the educational ana ^.dminiscrative, as well as 
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statistical, significance of results might be evaluated); 3) to examining 
not simply whether change occurs but also when it occurs (so that more 
tailored and effective programs and policies might be designed); 4) to 
exploring and measuring indirect, as well as direct, collegiate effects 
(so that the magnitude of college's effects will not be underestimated); 
5) to the study of college effects that may be conditional on student 
characteristics such as race/ethnicity, age, and sex (so that important 
variations in college's effects might be better understood and more 
effective programs and policies be designed); 6) to making greater use of 
qua"* cative research methods (so that important information inaccessible 
with quantitative methods will not be lost); 7) to expanding the 
theoretical perspectives that guide research and assessment study designs 
(so that theoretically myopic studies might be avoided), and 8) to 
focusing greater attention on the effects of the academic program and the 
teaching-learning process, the experiences of minority and older 
students, and the dynamics of students' interpersonal contacts with peers 
and faculty members (so that the educational experience might be 
maximized for all students )- 



FOOTNOTE 

1. Effect sizes can be estimated as the average change in 

freshman-to-senior year scores (whether for cross-sectional samples 
of, say freshmen and seniors, or for longitudinal panels) calculated 
in terms of standard deviation units- More specifically, an effect 
size can be estimated by subtracting a freshman year mean score from 
the senior year mean and then dividing that difference by the 
freshman year standard deviation. When expressed in standard 
deviation units, an effect size can be converted (using the area 
under the normal curve) to a "percentile point change" estimate - 
For example, given an estimated effect size eq^aal to 1 standard 
deviation, the area under the normal curve extends from the 50th to 
the 84th percentile, indicating a percentile point change of 34 
percentile points from the freshman to senior year. 

O 
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